This page discusses all of the storage options on HPC, as well as how to check to see how much space you have free in your home directory. First, a brief summary of the more commonly used storage options.

On our cluster, there are several dedicated file servers, usually referred to by the nickname "Thumpers." These servers are connected to HPC by the fast Infiniband network but the data on them is not backed up in any way. The Thumpers contain some redundant hardware so they can survive a limited amount of hardware failure without losing data. You also have a home directory, and the data in your home directory is backed up, and is moderately fast, but you can only store about 100 MB of data in your home directory. The data in your home directory or on the Thumpers is only accessible on the HPC cluster. The Thumpers are only intended for temporary file storage such as input and output files during computations.

For long-term storage, you have other options. The UMBC Research Data Storage network provides reliable data storage space, but it is slower than the Thumpers. You can rent that space yearly from OIT. Contact your PI to see if such space is already available. You can also access your UMBC-wide AFS partition, which is where your UMBC email and website, and your home directory on OIT lab machines are all stored. Your UMBC Research Data Storage and AFS files can be accessed anywhere in UMBC.

Table of Contents

How Quotas Work

Home directories are stored in a very small partition (only 200 GB) and so we have to limit how much space people can use. We limit two aspects of your storage: KB of disk space and the number of files you can create. For each of those quantities, you have two limits: a soft limit and a hard limit. When you reach your soft limit, you will have a little while to reduce your usage. If you wait too long before reducing your usage, or if you pass your hard limit, then you won't be able to make more files or enlarge existing files until you delete enough to get under your soft limit. Your hard and soft limits are also referred to as your quotas.

Currently, you only have quotas on your home directory, and those quotas are very low. You can store larger, temporary files elsewhere and there are other permanent storage options available which are described below.

How to Determine your Current Usage and Quotas

To see how much space you have left, run the quota program. Here is what the quota program prints out for my account:

Disk quotas for user straha1 (uid 29453): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
/dev/mapper/homevg-homelv
                  51380  100000  150000            3155   10000   15000        

Here's what each of those numbers mean, from left to right:

Your quotas will be the same. Of course your usage will be different (unless you deliberately conspire to have 3155 files that take up 51380 KB).

What to do When you Need More Space

Space Available to All Users

All users have access to the following storage locations. More details about these options can be found in the Initial Setup of Your Account page:

• home directory – the data space mentioned in the previous sections
• /tmp – a local storage area on each machine
• infiniband storage – fast but completely un-backed-up storage area
• research storage – slower, but more reliable storage area
• AFS – well-backed-up storage area accessible all over UMBC, and in other campuses as well

Other options may be available, depending on power and heating limitations.

The Local /tmp Directory

All nodes (including the login node hpc.rs.umbc.edu) have a local /tmp directory and it is generally 100 GB or larger in size. Any files in a machines /tmp directory are only visible on that machine and the system deletes them once in a while. Furthermore, the amount of storage space available in /tmp will vary wildly depending on how much data previous jobs have left behind. This directory should only be used while your job is running and you should be kind to other users and have your job delete any files it creates in /tmp.

14 TB Infiniband-connected Storage

This is a fast storage area that is not backed up – it is intended to be used as a temporary scratch space. Though it is not backed up, there is some built-in redundancy allowing the file servers to continue to function without loss of data if some hardware fails. If too much hardware fails simultaneously though, the file servers will lose data.

Generally, each Principal Investigator (PI) is given space in this storage area and that PI then splits it among their researchers. Usually this storage area can be accessed via the symbolic links ~/scratch/ and ~/common/ unless you deleted those symbolic links. The ~/scratch/ area is intended to be a scratch (temporary storage) space for your own files. The ~/common/ directory is for temporary files that you want to share between members of your research group.

The ~/scratch/ and ~/common/ are merely symbolic links to the 14 TB storage area – they are not actual directories. A symbolic link is like a pointer in C or a shortcut in Windows. It merely points to another file or directory or file. When you access a link in cd or another application, the application actually accesses the directory or file that the link points to. The ~/scratch/ symbolic link points to your scratch directory, and the ~/common/ symbolic link points towards your research group's shared storage area. Since those are symbolic links to directories, you can treat them as directories.

Data stored in ~/scratch/ and ~/common/ does not count towards your home directory quota. However, the filesystem on which your ~/scratch/ and ~/common/ data is stored has limited space. That space is shared by everyone who can write data to the filesystem (usually everyone in your research group). To determine how much space is free, run this command:

df -h ~/common/ ~/scratch/

For me, this prints out:

Filesystem            Size  Used Avail Use% Mounted on
thumper1-ib:/scratch/sparling
                      500G   44G  456G   1% /thumper1/scratch/sparling
thumper1-ib:/scratch/sparling
                      500G   44G  456G   1% /thumper1/scratch/sparling

You will get a similar message, but with different directory names and sizes. The 500GB 44GB 456GB lines tell you the storage limit, how much storage has been used, and how much space is still available. The first line that lists sizes is for ~/common/ (since you listed ~/common/ first in the df command) and the second line is for ~/scratch/. Note that there are no file limits – you can make as many files as you want so long as you don't use up all of the storage space. The filesystem itself has limits as to how many files and directories you can create, but those limits are usually very large.

The thumper1-ib:/scratch/sparling tells you where the data is physically stored. The thumper1-ib is the name of the remote machine that stores the data, and the /scratch/sparling tells you where on that machine the data is stored. The /thumper1/scratch/sparling tells you where that directory is mounted on the local machine. In other words, /scratch/sparling on thumper1-ib is accessible from /thumper1/scratch/sparling on the machine on which you ran df. Note that, in my case, the ~/scratch/ and ~/common/ directories are stored on the same filesystem (they are both on thumper1-ib:/scratch/sparling). Thus, any files I put in my ~/scratch/ or ~/common/ will count towards the same storage limit. In your case, those directories might be on different filesystems. If so, data stored in one directory will not count towards the storage limit of the other.

If you do not know where your Inifiniband-connected storage area is, contact your PI or user support.

Research Data Storage

This is a reliable storage area that can be accessed anywhere in UMBC, including from the HPC head node. Any UMBC researchers can purchase research data storage space. The file servers that make up this storage network each have redundant hardware. Thus no file server will suffer loss of data if there are minor hardware failures. If a file server has too many hardware failures at the same time, it will lose data. However, all data is stored twice – once in servers in the Public Policy building and once in servers in Engineering. Thus, even if an entire file server or entire server room fails, your data will not be lost. Also, we store ten daily archives of the data so if you accidentally delete data, we can undelete it for you. (The archives take very little space since the filesystem just stores the changes during each 24-hour period.)

The research data storage is accessible via NFS (a unix file sharing protocol). Thus you can mount this storage area on your local machines. In other words, you can easily set up your machine to make your research data storage accessible as if it was locally stored on your machine. The storage area can also be shared by Samba which allows it to be shared in that manner with a Windows machine as well. Samba file sharing is not yet tested but it should work. Currently, the research data storage is not accessible on the cluster nodes but it is accessible on the head node. If you need it to be accessible on the cluster nodes, that can be arranged.

Any researchers at UMBC can purchase research data storage space for a cost of $200 per year per 100 GB. You are billed every year on July 1. If you purchase storage after July 1, the charge will be prorated to July 1 (if you purchase mid-year you only pay for the remaining portion of the year).

If you wish to purchase research data storage space, or for any other inquiries related to the research data storage area, contact our research data storage point of contact.

AFS

Your AFS partition is the directory where your personal files are stored when you use the OIT computer labs or the gl.umbc.edu login nodes. The UMBC-wide /afs can be accessed on HPC via the /afs directory. Your AFS partition is in /afs/umbc.edu/u/s/username where username is your username, u is the first letter of your username and s is the second letter. Your /afs/umbc.edu/u/s/username/pub directory is where you store files visible to other users, such as your umbc webpage in pub/www/. Your /afs/umbc.edu/u/s/username/home directory is your AFS home directory.

In order to access AFS, you need an AFS token. One is given to you when you log in, but it expires a few hours after you log in. You can see whether you currently have an AFS token by running the tokens command:

straha1@hpc:~> tokens

Tokens held by the Cache Manager:

Tokens for afs@umbc.edu [Expires Oct 25 00:16]
   --End of list--

The "Tokens for afs@umbc.edu" line tells me that I currently have tokens that let me access UMBC's AFS storage. The expiration date ("Expires Oct 25 00:16") tells me when my tokens will expire. When your tokens expire, you will get this message:

straha1@hpc:~> tokens

Tokens held by the Cache Manager:

   --End of list--

Notice the lack of the "Tokens for afs@umbc.edu" line. Once your tokens expire, you have to get new ones in order to access AFS again. To do that, use the aklog command:

straha1@hpc:~> aklog
straha1@hpc:~> tokens

Tokens held by the Cache Manager:

Tokens for afs@umbc.edu [Expires Oct 25 00:28]
   --End of list--

Other Storage Options

Depending on heating, power and other logistical limitations, you may be able to make other storage arrangements with OIT. Some groups have already made other arrangements – contact your PI or user support to find out if your group has other such storage spaces. If you are interested in inquiring about obtaining such storage, contact user support.